For centuries, printed visual displays have been used for displaying information in the form of bar charts, pie charts and graphs. In the information age, visual displays (e.g., computer monitors) have become the primary means for conveying large amounts of information. Computers with visual displays, for example, are often used to process and/or monitor complex numerical data such as financial trading market data, fluid flow data, medical data, air traffic control data, security data, network data and process control data. Computational processing of such data produces results that are difficult for a human overseer to monitor visually in real time. Visual displays tend to be overused in real-time data-intensive situations, causing a visual data overload. In a financial trading situation, for example, a trader often must constantly view multiple screens displaying multiple different graphical representations of real-time market data for different markets, securities, indices, etc. Thus, there is a need to reduce visual data overload by increasing perception bandwidth when monitoring large amounts of data.
Sound has also been used as a means for conveying information. Examples of the use of sound to convey information include the Geiger counter, sonar, the auditory thermometer, medical and cockpit auditory displays, and Morse code. The use of non-speech sound to convey information is often referred to as auditory display. One type of auditory display in computing applications is the use of auditory icons to represent certain events (e.g., opening folders, errors, etc.). Another type of auditory display is audification in which data is converted directly to sound without mapping or translation of any kind. For example, a data signal can be converted directly to an analog sound signal using an oscillator. The use of these types of auditory displays have been limited by the sound generation capabilities of computing systems and are not suited to more complex data.
Sonification is a relatively new type of auditory display. Sonification has been defined as a mapping of numerically represented relations in some domain under study to relations in an acoustic domain for the purposes of interpreting, understanding, or communicating relations in the domain under study (C. Scaletti, “Sound synthesis algorithms for auditory data representations,” in G. Kramer, ed., International Conference on Auditory Display, no. XVIII in Studies in the Sciences of Complexity, (Jacob Way, Reading, Mass. 01867), Santa Fe Institute, Addison-Wesley Publishing Company, 1994.). Using a computer to map data to sound allows complex numerical data to be sonified.
Applications of sonification have been most common in the medical field, for example, as disclosed in U.S. Pat. Nos. 6,449,501; 6,283,763; 6,083,163; 5,836,302; and 5,730,140, which are incorporated herein by reference. Applications of sonification have been limited in other fields. One example of sonification applied to stock market data is disclosed in U.S. Pat. No. 5,371,854 to Kramer, which is incorporated herein by reference. Kramer discloses a sonification system using auditory beacons as references for comparison and orientation in data. Other attempts at sonification of stock market data include work by Keith Nesbitt and Stephen Barrass published on Jul. 2-5, 2002 and entitled “Evaluation of a Multimodal Sonification and Visualization of Depth of Stock Market Data,” given at the Proceedings of ICAD, Tokyo, 2002, in which vocalized alerts were used to indicate “buy” and “sell” in a trading system. A work by Jonathon Berger of CCRMA, Stanford, published on the web at Stanford University at http://www-ccrma.stanford.edu/groups-/soni/index.html, discloses sonification of historical financial data using pulses of filtered noise, one pulse for each trading day.
The human ability to recognize patterns in sound presents a unique potential for the use of auditory displays. Patterns in sound are recognized over time, and a departure from a learned pattern results in an expectation violation. For example, individuals establish a baseline for the “normal” sound of a car engine and can detect a problem when the baseline is altered or interrupted. Also, the human brain can process voice, music and natural sounds concurrently and independently. Music, in particular, has advantages over other types of sound with respect to auditory cognition and pattern recognition. Musical patterns are implicitly learned, recognizable even by non-musicians, and aesthetically pleasing.
The existing auditory displays have not exploited the true potential of sound, and particularly music, as a way to increase perception bandwidth when monitoring data. U.S. Pat. No. 5,371,854, for example, does not disclose a sonification that is based on music and specifically common-practice music. Kramer relies solely on the use of beacons to discern trends in data rather than the ability to recognize musical patterns. Thus, existing sonification techniques do not take advantage of the auditory cognition attributes unique to music.
Accordingly, there is a need for a musical sonification system and method that is capable of increasing data perception bandwidth by using musical patterns.